Exploring the limits of system-directed dialogue, dialogue evaluation of the danish dialogue system

Spoken language dialogue systems technologies are beginning to master the design and implementation of applied systems for complex well-structured tasks. Partly for this reason, there is a need for evaluation metrics which include general concepts of task and dialogue types. The paper reports on the scenario-based user test of the dialogue management of an airline ticket reservation system. The test data are compared to the data from the last Wizard of Oz iteration before the system was implemented. Detailed analysis of user dialogue behaviour reveals a series of principled limitations of systemdirected dialogue for complex well-structured tasks. The discussion weighs those limitations against the demonstrated potential of system-directed dialogue for a broad class of tasks.